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EXECUTIVE  SUMMARY 


Requirement: 

To  develop  the  Job  Skills  Education  Program  (JSEP)  evaluation  plan,  which 
should  establish  whether  improved  performance  on  criterion  measures  was  due  to 
the  new  program.  The  contractor  shall  select  or  develop  evaluation  instruments 
and  develop  a  data  collection  scheme  which  will  reflect  evaluation  issues  as 
well  as  quality  control  and  revision. 

Procedure: 

The  Florida  State  University  JSEP  internal  evaluation  team  developed  an 
evaluation  plan  based  on  a  comprehensive  study  of  the  JSEP  mission  from  which 
objectives  (both  formative  and  summative)  were  defined  and  delineated. 

Criterion  measures  were  then  either  selected  or  developed  based  on  the 
delineated  objectives. 

The  evaluation  plan  which  consists  of  objectives,  evaluation  instruments 
and  data  collection  procedures,  has  undergone  extensive  internal  and  external 
reviews.  Revisions  were  made  to  the  original  plan  based  on  these  reviews. 

Findings: 

An  evaluation  plan  consisting  of  objectives,  evaluation  instruments,  and 
data  collection  procedures  constitute  the  findings  of  this  task.  The 
evaluation  plan  is  divided  into  two  major  parts:  the  evaluation  of  the 
preliminary  tryout  of  JSEP  at  Fort  Rucker  in  1984,  and  the  full  scale  tryout  of 
JSEP  at  four  sites  in  1985. 


Use  of  Findings: 

The  evaluation  plan  will  be  implemented  during  the  preliminary  and 
full  scale  tryouts.  The  data  will  provide  the  information  required  to  assess 
the  effectiveness  of  the  JSEP  system. 


FOREWORD 


The  Job  Skills  Education  Program  (JSEP)  is  a  multi-phase  program  begun  in 
Fiscal  Year  1982,  and  designed  to  enhance  enlisted  career  potential  by 
improving  soldier  job  performance.  The  sponsor,  the  Education  Division, 

Office  of  the  Deputy  Chief  of  Staff  for  Personnel,  expects  JSEP  to  replace 
the  Army's  current  Basic  Skills  Education  Program  when  it  is  implemented. 

The  JSEP  program,  being  developed  by  Florida  State  University  (FSU)  will 
result  in  a  standardized  curriculum  for  soldiers  who  demonstrate  deficiencies 
in  the  knowledge  and  skills  required  to  successfully  learn  their  Military 
Occupational  Specialty  (MOS) . 

In  accordance  with  current  policy,  JSEP  will  be  an  on-duty  program.  It  will 
also  use  a  computer-based  management  system  to  facilitate  an  open  entry/open 
exit  approach.  At  present,  most  of  the  lessons  being  developed  will  be 
computer  delivered;  however,  the  plan  calls  for  using  existing  materials,  and 
incorporating  materials  developed  as  part  of  other  ARI  efforts,  whenever 
appropriate. 

A  unique  aspect  of  JSEP  is  that  it  builds  upon  a  very  detailed  front-end 
analysis  of  MOS  Baseline  Skills.  The  analysis  covered  tasks  performed  by 
soldiers  in  the  94  highest  density  MOSs,  in  addition  to  Common  Tasks  (the 
skills  that  all  soldiers,  regardless  of  their  MOS,  need  to  know).  Although  the 
Army  has  over  300  MOSs,  the  94  covered  in  the  analysis  represent  about  80%  of 
all  soldiers.  Perhaps  the  most  useful  product  developed  for  the  analysis  was 
a  taxonomy  listing  more  than  200  prerequisite  competencies. (P.C. )  for  these 
MOSs.  The  competencies  were  derived  from  detailed  reviews  of  Soldier  Manuals, 
and  from  extensive  interviews  with  subject-matter  experts  at  Army  schools. 

This  effort  produced  a  series  of  tests  intended  to  diagnose  deficiencies  in 
the  P.C.s.  Modified  versions  of  these  tests  will  be  used  in  JSEP. 

The  JSEP  program  will  include  a  front-end  learning  strategies  module 
designed  to  improve  soldier  skills  in  reading,  studying,  test  taking,  and 
problem  solving.  The  curriculum  will  consist  of  this  strategies-training,  plus 
180  diagnostic  review  lessons,  and  120  skill  development  lessons,  which  are 
being  developed  for  the  PLATO  and  MicroTICCIT  computer  systems.  The  program  is 
being  tried  out  at  two  TRADOC  sites  and  two  F0RSC0M  sites,  prior  to  an  Army-wide 
phased  implementation. 


JSEP:  THE  EVALUATION  PLAN 


OVERVIEW 


Operational  Problem 

Soldiers  must  be  trained  so  that  each  Army  job  is  performed  competently— 
regardless  of  differences  in  the  ability  and  background  of  new  enlistees.  To 
accept  a  lower  performance  level  would  cause  many  mission  elements  to  fail. 

Many  Army  jobs  are  increasingly  dependent  upon  the  soldiers'  abilities  to 
use  high  technology  and  to  learn  new  technology  as  it  develops.  Soldiers  need 
more  than  training.  They  need  enough  education  to  learn  subsequent  jobs,  to 
become  eligible  for  promotion,  and,  ultimately,  to  provide  leadership  for 
tomorrow's  Army. 

The  Job  Skills  Education  Program  (JSEP)  is  primarily  designed  to  provide 
soldiers  with  job-related  basic  skills  instruction  prerequisite  to  learning 
skill  level  1  and  2  job  tasks  during  first  duty  assignments.  An  extensive  job 
analysis  by  RCA  Educational  Services  of  the  94  most  populous  Military 
Occupational  Specialties  (MOS)  and  tasks  contained  in  the  Soldier's  Manual  of 
Common  Tasks  identified  MOS  specific  indicators  of  functional  basic  skills 
required  in  each  MOS.  These  are  the  skills  which  will  be  taught  in  JSEP. 

The  vast  majority  of  soldiers  have  been  exposed  to  basic  skills  instruc¬ 
tion  before  entering  the  Army.  However,  many  entering  soldiers  have  not 
learned  or  retained  those  basic  skills  well  enough  to  apply  them  to  learning 
more  advanced  skills.  To  help  soldiers  learn  better  and  remember  more,  JSEP 
incorporates  straightforward  training  in  job  related  basic  skills  along  with 
instruction  on  research-based  learning  strategies  that  are  directly  aimed  at 
learning  and  retaining  job  skills. 


Research  Objective 

The  purpose  of  Task  7  is  to  describe  the  JSEP  evaluation  plan  in  terms  of 
objectives,  procedures,  and  data  collection  instruments  to  be  used  in  the  Task 
13  preliminary  tryout  (to  be  held  at  Fort  Rucker)  and  the  Task  16  full  scale 
tryout  (tentatively  scheduled  at  Forts  Leonard  Wood,  Riley,  Bliss,  and  Lewis). 
The  report  is  organized  into  two  parts.  Part  I  describes  the  evaluation  plan 
for  the  preliminary  tryout  during  Phase  II  which  seeks  to  test  the 
effectiveness  of  typical  lesson  designs,  as  well  as  to  assess  early 
configurations  of  the  JSEP  instructional  system.  Part  II  describes  the  plan 
for  the  full  scale  tryout  during  Phase  III  which  seeks  to  assess  the 
effectiveness  of  JSEP  in  helping  soldiers  master  job  related  prerequisite 
competencies  (PCs),  to  relate  the  mastery  of  PCs  to  simulations  of  job 
performance,  and  to  investigate  the  potential  influence  of  JSEP  in  maximizing 
soldier  performance  on  career-determining  examinations. 


Preliminary  drafts  of  the  evaluation  plan  have  undergone  review  by  The 
Florida  State  University  (FSU)  and  Hazeltine  internal  evaluation  review 
committee,  Warren  Simmons  and  Gail  Baker  of  the  Army  Research  Institute  (ARI), 
and  Clifford  Hahn  of  the  American  Institutes  for  Research.  Additionally,  Gary 
Peterson  and  Richard  Kraft  of  FSU,  and  Lois  Wilson  and  Boyd  Richards  of 
Hazeltine  reviewed  the  elements  of  the  evaluation  plan  pertaining  to  the 
preliminary  tryout  with  Frank  Barbour,  Assistant  to  the  Education  Services 
Officer  (ESO)  at  Fort  Rucker. 

The  FSU-Hazeltine  team  realizes  that  there  will  be  additional  refinements 
in  the  plan  as  it  moves  from  concept  to  operation.  However,  looking  to  the 
future,  we  believe  that  the  following  issues  must  be  given  priority 
consideration:  (1)  enlisting  command  support  to  provide  soldiers 
representative  of  the  intended  JSEP  target  population,  (2)  obtaining  ESO 
support  to  provide  adequate  physical  facilities,  and  (3)  establishing  workable 
arrangements  among  FSU,  ARI,  TRAOOC,  and  the  Fort  Rucker  Education  Center  for 
personnel  for  the  preliminary  tryout,  and  with  FORSCOM  for  the  full-scale 
tryout. 

An  outcome  of  an  approved  evaluation  plan  should  be  a  list  of  requirements 
to  implement  JSEP  at  the  respective  tryout  sites.  Contractual  arrangements  and 
responsibilities  can  then  be  established  among  FSU,  ARI,  TRADOC,  FORSCOM,  and 
the  local  education  centers  to  meet  these  requirements. 


THE  JSEP  PRELIMINARY  TRYOUT 

The  purpose  of  Task  13,  Conduct  Preliminary  Tryout,  is  to  identify 
potential  problems  in  selected  initial  drafts  of  JSEP  instructional  materials. 
Additionally,  the  preliminary  tryout  will  collect  data  on  certain  elements  of 
the  JSEP  instructional  system  in  an  operational  environment.  It  is  planned 
that  the  Task  13  preliminary  tryout  will  be  conducted  at  Fort  Rucker,  Alabama, 
using  both  PLATO  and  TICCIT  systems. 

Objectives 

The  objectives  of  the  preliminary  (Task  13)  tryout  are  as  follows: 

1.  Find  out  how  quickly  and  how  well  soldiers  are  able  to  use  JSEP  as 
designed,  and  obtain  feedback  on  lessons  from  soldiers  and  education  center 
personnel.  The  preliminary  tryout  will  provide  an  indication  of  the 
effectiveness  of  the  FSU  in-house  review  process  used  to  inspect  lessons  prior 
to  soldier  testing. 

2.  Identify  implementation  issues  for  selected  components  of  the  JSEP 
instructional  delivery  system  including  hardware  (PLATO  and  TICCIT), 
orientation  procedures,  diagnostic  prescriptive  tests  (Locator  and  lesson 
pretests),  selected  lessons,  criterion-referenced  posttests,  personnel  (tutors 
and  ESO),  the  management  information  system,  and  the  evaluation  instruments. 

The  tryout  should  provide  a  good  estimate  of  staffing  arrangements 
necessary  to  conduct  the  full  scale  tryout. 
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3.  Examine  the  degree  to  which  the  Locator  Tests  predict  performance  on 
selected  individual  lesson  pretests.  The  Locator  and  lesson  pretests  are 
designed  tc  measure  the  same  ability,  i.e.,  the  soldiers'  proficiency  levels  on 
PCs.' 


4.  Assess  soldier  attitudes  about  using  JSEP.  JSEP  effectiveness  will 
depend  largely  on  the  motivation  and  effort  of  soldiers  to  master  their  assigned 
lessons  within  planned  time  limits.  The  completion  rate  may  be  influencea  by 
solaier  attitudes  related  to  the  ambience,  instructional  approach,  human 
factors,  and  content  of  the  lessons.  Thus,  the  evaluation  instruments  used  in 
the  preliminary  tryout  will  provide  an  indication  of  the  attitudes  of  soldiers 
toward  various  elements  of  the  JSEP  system. 

5.  Investigate  the  possible  instructional  requirements  for  enabling 
soldiers  to  pass  the  summative  posttest.  The  summative  posttest  will  be 
composed  of  36  items  (three  items  randomly  drawn  from  each  of  the  12  lessons 
soldiers  will  have  completed).  Correlations  between  the  lesson  posttests  and 
the  summative  posttest  will  be  examined  in  order  to  investigate  the  utility  of 
this  procedure  for  developing  future  summative  posttests. 

An  index  of  classification  efficiency  (albeit  crude)  will  be  derived  for 
each  lesson  included  in  the  tryout  to  ascertain  concurrent  validity.  If  the 
classification  correlations  are  low,  the  JSEP  testing  system  components  for 
routing  the  learner  to  appropriate  lessons  will  be  examined. 

It  could  be  that  some  soldiers  may  be  able  to  pass  the  summative  posttest 
without  having  completed  all  of  the  assigned  lessons,  while  other  soldiers  may 
have  difficulty  passing  this  exam  even  though  they  have  passed  ill  of  their 
assigned  individual  lesson  posttests.  An  analysis  will  be  made  of 
relationships  between  performances  on  the  individual  lesson  po'ttests  and  the 
summative  posttest. 


Curriculum  for  the  Prel iminary  Tryout 

The  tryout  curriculum  will  be  divided  into  two  segments:  (1)  a  block  of 
instructional  events  simulating  the  complete  JSEP  curriculum  and  (2)  a  block 
of  additional  selected  short  lessons.  The  events  of  the  simulated  JSEP  (see 
Figure  1)  include  the  following: 

o  Orientation  This  element  includes  introductions  of 

participants  and  staff,  an  overview  of  JSEP 
and  its  purposes,  the  completion  of  a 
background  information  survey,  and  a 
demonstration  of  how  to  operate  the  computer 
based  instruction  (CBI)  system. 

o  Locator  Tests  These  tests,  two  30  item  multiple-choice 

verbal  and  mathematics  tests,  will  be 
administered. 
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2.5  hrs  I  Orientation  (JSEP,  background,  computer  usaoe)  classroom 


1.5  hrs  Locator  (paper  and  pencil  administered  in  classroom) 

2  hrs  Learning  Strategies: 

I _ Intro  to  Memory  (on-line)  _ 


8  hrs  Short  lessons  (12),  each  with  lesson  pretest  and 

posttest 


8  hrs 


Long  Lessons  (4) 

with  applications  in  memory  strategies 

(a)  Capitals 

(b)  Rounding 

(c)  Gauges 

(d)  Addition 


Summative  Posttest,  CLOZE  Test 
Attitude  Survey  and  interview 


Total  24  hrs 


Additional  Formative  Evaluation 


As  time 
permits 


Selected  Short  lessons 


Figure  1.  Curriculum  structure  for  JSEP  tryout. 


o  Learning  Strategies 
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o  Short  Lessons 


o  Long  Lessons 


o  Summative  Posttest 


These  will  be  administered  prior  to  the 
short  lessons.  A  lesson  on  mood 
management  will  be  presented.  Selected 
techniques  such  as  memory  strategies, 
relaxation,  and  positive  self-talk, 
will  be  included. 

Twelve  short  lessons  selected  on  the  basis 
of  breadth  of  curriculum  and  learning 
approaches  will  be  delivered  by  computer. 
Each  short  lesson  consists  of  a  pretest,  a 
brief  review  of  essential  rules,  concepts, 
and  operations,  a  short  practice,  and  a 
posttest.  The  purpose  of  the  short  lessons 
is  to  take  advantage  of  prior  learning  to 
reduce  learning  time  and  cost.  We 
anticipate  that  many  soldiers  may  initially 
fail  to  demonstrate  mastery  of  a  PC  on  the 
lesson  pretest  but,  with  a  brief 
"refresher,"  they  will  be  able  to  pass  the 
posttest.  For  the  preliminary  trial, 
however,  all  soldiers  will  take  all  12 
short  lessons. 

Four  complete  long  lessons  will  be 
delivered  by  computer  which  correspond  to 
four  of  the  twelve  short  lessons.  Each  of 
these  lessons  will  include  instruction, 
practice,  help,  and  a  posttest.  Lessons  are 
described  more  fully  in  the  Task  5  Report 
for  this  project.  Again,  for  the 
preliminary  trial,  all  soldiers  will  take 
the  4  long  lessons  regardless  of  pretest 
scores. 

All  participants  will  take  a  36  item 
summative  posttest  on  the  assigned  lessons. 
In  the  full  scale  tryout,  soldiers  will 
normally  take  the  summative  test  only  on  the 
short  and  long  lessons  to  which  they  have 
been  assigned.  For  the  preliminary  tryout, 
the  items  will  be  drawn  from  the  posttests 
of  the  12  lessons  included  in  the  JSEP 
simul ation. 


o  Performance  Test  A  CLOZE  Test  will  also  be  included  as  a 

pilot  test  of  the  performance  test  to  be 
included  in  the  full  scale  tryout.  The 
CLOZE  Test  is  often  used  to  measure 
readability  level  of  materials  but  can  also 
be  used  to  measure  reading  comprehension. 
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At  Fort  Lewis,  the  CLOZE  Test  was  used  as  a 
measure  of  the  degree  to  which  BSEP  II 
soldiers  could  read  job  related  field 
manuals  (Banner,  Ridge,  Hewitt,  &  Christ, 
1982). 

Additionally,  items  will  be  selected  from 
the  RCA  criterion  tests  on  the  basis  of 
their  validity  as  measures  of  skills  taught 
in  the  12  lessons.  The  items  are  of  short 
answer  variety  and  require  soldiers  to 
formulate  their  own  responses  to  job-like 
scenarios.  We  believe  such  a  tect  will 
assess  a  soldier's  ability  to  recall  and 
apply  knowledge  and  skills  acquired  from  the 
JSEP  lessons. 

o  JSEP  Attitude  Survey  An  attitude  survey  will  be  administered 

after  the  soldiers  complete  the  summative 
and  CLOZE  Tests.  The  attitude  survey 
will  cover  a  variety  of  aspects  of  JSEP 
and  computer  based  instruction. 

A  second  set  of  selected  short  lessons  will  be  available  for  solders  who 
complete  the  JSEP  simulation  before  they  are  scheduled  to  return  to  duty. 

These  lessons  will  be  selected  for  “opportune"  formative  evaluation  on  the 
basis  of  their  representativeness  of  the  entire  JSEP  curriculum. 


Physical  Facilities 

The  learning  center  at  Fort  Rucker  currently  has  4  PLATO  terminals  in 
operation.  Education  center  personnel  indicated  that  we  will  be  able  to  use 
these  terminals  for  the  preliminary  tryout.  Four  TICCIT  terminals  will  also 
be  installed  in  the  education  center.  The  room  has  adequate  electricity  and 
air  conditioning  but  will  need  carrels  or  tables  and  chairs.  A  formal  thirty 
seat  classroom  is  also  available  for  orientation  and  testing.  Frank  Barbour, 
the  assistant  ESO  at  Fort  Rucker,  has  been  informed  of  the  preliminary  tryout 
implementation  date. 


METHOD:  THE  PRELIMINARY  TRYOUT 

The  Soldiers 

The  preliminary  tryout  will  require  48  soldiers.  To  obtain  a  sample  of 
soldiers  typical  of  the  JSEP  population  entails  enlisting  the  support  of 
Commanding  Officers  (COs).  The  soldiers  considered  for  JSEP  should  be  those 
who  have  recently  arrived  at  their  duty  station  (within  3  months)  and  have 
earned  General  Technical  (GT)  scores  on  the  ASVAB  between  80  and  99.  This 
group  would  much  resemble  a  typical  Basic  Skills  Education  Program  (BSEP  II) 
population  who  enroll  in  Basic  Skills  Instruction  to  raise  test  scores  (e.g., 
GT,  General  Educational  Development  [GED],  Skill  Qualification  Test  [SQT]), 
but  who  may  or  may  not  have  difficulty  in  learning  their  job  tasks.  A  data 


file,  with  6T  scores  and  educational  attainments,  is  already  available  from  the 
Fort  Rucker  ESC.  Once  the  names  are  drawn  (at  random)  from  such  a  file,  the 
COs  will  be  reauested  by  the  £S0  to  release  soldiers  from  duty  at  a  mutually 
convenient  time. 

For  the  preliminary  trial,  individuals  from  the  FSU-Hazeltine  team 
will  be  designated  to  serve  as  JSEP  coordinators  to: 

1.  assist  the  soldiers  in  registration  procedures, 

2.  conduct  the  orientation  to  JSEP, 

3.  teach  the  soldiers  how  to  use  the  computer,  and 

4.  provide  instructional  support  in  the  event  that 
individuals  are  unable  to  continue  with  their 
assigned  lessons. 

JSEP  coordinators  will  be  instructed  not  to  teach  the  lessons  to  the 
soldiers  or  to  provide  excessive  coaching  but  to  provide  assistance  mainly  when 
soldiers  have  difficulty  operating  the  computer. 

The  main  goal  of  the  preliminary  tryout  is  to  identify  parts  of  lessons 
that  may  not  be  self-instructional  and  to  determine  plausible  causes.  This 
information  will  be  used  to  revise  the  lessons  and  to  improve  the  F5U  lesson 
design  and  review  procedures.  An  outside  observer,  preferably  from  the 
education  center,  should  collect  information  in  order  to  define  the  role  of 
JSEP  coordinators  so  that  training  can  be  revised,  as  needed,  for  the  Task  16 
tryout. 


Procedures 

Soldiers  will  be  asked  to  report  to  the  education  center  from  Friday 
of  one  week  through  the  Friday  of  the  next  week  for  four  hours  each  day.  Each 
will  be  assigned  to  the  center  for  24  instructional  hours.  The  tryout  will 
include  two  cohort  groups,  one  assigned  one  week  and  the  other  assigned  the 
following  week.  Each  cohort  will  have  a  morning  and  afternoon  group  of  12  each. 

On  the  first  Friday,  soldiers  will  complete  a  background  information 
survey.  The  JSEP  coordinator  will  then  explain  the  JSEP  system  and  the 
purpose  of  the  tryout.  A  brief  JSEP  orientation  program  will  be  included 
here.  The  JSEP  coordinator  will  inform  soldiers  of  the  rules  and  procedures 
of  JSEP  operation,  procedures  to  follow  when  they  finish  the  JSEP  simulation 
segment  of  the  tryout  and  when  to  take  breaks.  Participants  will  then  be 
instructed  in  the  operation  of  the  relevant  CB1  system  (PLATO  or  TICCIT). 

During  the  orientation  period,  the  soldiers  will  also  take  the  Locator  Test  and 
a  pretest  version  of  the  summative  posttest. 

The  following  Monday,  the  soldiers  will  initially  report  to  the 
orientation  room  for  the  performance  test.  They  will  then  go  to  the  learning 
center  and  will  be  assigned  to  either  a  TICCIT  or  PLATO  terminal  for  the  trial 
period.  Alternates  will  take  the  paper  version  of  JSEP.  The  introductory  unit 
on  learning  strategies  will  be  taken  first.  As  soloiers  complete  the  learning 


Soldiers  are  expected  to  progress  through  the  short  lessons  at  their  own 
rates.  Soldiers  will  also  take  four  long  lessons  which  accompany  four  of  the 
twelve  short  lessons.  When  soldiers  finish  each  long  lesson,  they  will 
complete  a  Long  Lesson  Questionnaire  (see  Appendix  A)  for  that  lesson.  When 
soldiers  are  unable  to  continue  in  any  lesson,  they  will  be  instructed  to 
request  assistance  from  a  OSEP  coordinator. 

On  the  last  Friday,  when  soldiers  have  completed  their  twelve  short 
lessons  and  four  long  lessons,  they  will  be  given  a  summative  posttest  over 
all  lessons.  This  test  will  be  composed  of  36  items  drawn  from  the  posttests 
of  the  twelve  lessons  comprising  the  tryout.  When  the  soldiers  have  completed 
the  summative  posttest,  they  will  take  a  CLOZE  Test  and  the  RCA  criterion  test. 

The  JSEP  General  Questionnaire  (see  Appendix  B)  will  then  be  administered 
to  assess  their  attitudes  regarding  their  experience  in  the  learning  environ¬ 
ment.  When  the  soldiers  complete  their  assigned  lessons,  the  summative  post¬ 
test,  the  CLOZE  Test,  the  RCA  criterion  test,  and  the  JSEP  General 
Questionnaire,  they  will  return  to  duty.  A  JSEP  coordinator  will  carry  out  the 
required  check-out  procedures  to  insure  that  all  required  information  has  been 
secured. 


The  JSEP  Instruments 

The  following  instruments  will  be  used  to  gather  information  pertinent  to 
the  objectives  of  the  tryout  and  copies  are  located  in  the  Appendixes. 

1.  JSEP  Soldier  Background  Information  Survey.  This  survey  will  request 
information  from  each  participant  pertaining  to  their  military  history  and 
familiarity  with  CBI.  It  will  be  administered  during  the  initial  orientation 
procedures.  The  information  will  be  useful  for  interpreting  soldier  achieve¬ 
ment  levels  and  attitudes  (see  Appendix  C). 

2.  Resource  Utilization  Survey:  Learning  Center  Operation.  This  survey 
will  be  completed  by  an  FSU  or  Hazeltine  observer  stationed  in  the  learning 
center.  It  will  contain  such  information  as  no-show  rates,  computer  down-time, 
and  total  minutes  soldiers  were  interrupted  due  to  environmental  disturbances 
(see  Appendix  0). 

3.  JSEP  Learning  Outcome  Form,  The  JSEP  Learning  Outcome  Form  will 
contain  test  scores  earned  by  the  soldiers.  Test  scores  include  those  from 
the  Locator  Test  (Verbal  and  Quantitative),  lesson  pretests,  lesson  posttests, 
the  summative  posttest  and  the  CLOZE  test.  During  the  preliminary  tryout, 
this  information  will  be  gathered  on-line  (see  Appendix  E). 

4.  Long  Lesson  Questionnaire.  At  the  completion  of  each  long  lesson 
soldiers  "will  complete  a  short  questionnaire  related  to  the  quality  of 
instruction.  This  questionnaire  will  assess  certain  human  factors  as  well  as 
instructional  quality  dimensions,  such  as  pacing,  sequence,  exercises;  it  will 
also  solicit  suggestions  for  improvements,  (see  Appendix  A). 


5.  JSEP  General  Questionnaire.  This  questionnaire  will  assess  attitudes 
of  soldiers  toward  various  aspects  of  JSEP.  It  addresses  three  areas:  (1) 
computer  based  instruction,  (2)  the  JSEP  curriculum,  and  (3)  general  questions 
concerning  JSEP  in  the  Army  (see  Appendix  B). 

6.  JSEP  Screen  Display  Questionnaire.  This  instrument  is  designed  to 
assess  soldiers'  preferences  for  information  presentation  on  the  screen. 


Evaluation  Standards  and  Design 

For  the  tryout,  arbitrary  minimum  passing  scores  will  be  set  for  the 
lesson  posttests  and  the  summative  posttest.  The  group  who  will  set  the 
minimum  scores  for  the  tryout  lessons  will  include  available  subject  matter 
experts,  instructional  designers,  and  teachers  from  the  Fort  Rucker  BSEP  II 
classes. 

Individual  Lesson  Standard.  Three  standards  are  appropriate  for  the 
tryout  for  the  first  set  of  lessons  (twelve  short  and  four  long):  effective¬ 
ness,  efficiency,  and  attitude.  The  effectiveness  standard  will  be  based  upon 
the  percentage  of  soldiers  who  fail  the  lesson  pretest  but  pass  the  first 
trial  of  a  lesson  posttest  a*ter  taking  the  short  and  long  lessons.  The 
efficiency  standard  will  be  the  median  amount  of  time  to  complete  each  lesson 
(T^e,  time  to  reach  a  passing  score).  The  median  is  used  since  the 
distributions  of  completion  times  among  learners  is  likely  to  be  highly 
skewed.  Attitudes  toward  the  four  long  lessons  will  be  assessed  by  the  Long 
Lesson  Questionnaire  and  attitudes  toward  the  JSEP  environment  will  be  assessed 
by  the  JSEP  General  Questionnaire. 

With  the  remaining  time,  as  many  short  lessons  as  possible  will  be  field- 
tested  on  the  soldiers.  Using  these  supplemental  lessons  during  the  Ft. 

Rucker  tryout  will  provide  an  additional  indication  of  how  much  additional 
field  testing  will  be  required  for  formative  evaluation  of  the  lessons. 

JSEP  System  Standard.  The  evaluation  standards  used  for  the  JSEP  system 
include: 

1.  effectiveness, 

2.  efficiency,  and 

3.  attitudes  toward  the  experience. 

The  effectiveness  criterion  will  be  assessed  by  recording  the  number  of 
soldiers  who  pass  the  short  and  long  lessons  on  the  first  trial.  The 
efficiency  criterion  will  be  the  median  amount  of  time  to  complete  the  assigned 
lessons  and  the  pass  or  fail  rate  on  the  lesson  posttests  and  on  the  summative 
posttest.  Attitudes  toward  JSEP  will  be  assessed  by  the  JSEP  General 
Questionnaire. 

The  Evaluation  Design.  The  basic  design  for  the  JSEP  simulation  in 
preliminary  tryout  is  a  one  group  pretest-posttest  design  as  diagrammed  in 
Figure  2. 

While  such  a  design  lacks  internal  validity  due  to  history,  maturation, 
and  testing  effects,  we  believe  that  with  sufficient  data  at  the  observation 
points  and  with  statistical  controls  through  covariance,  effective  formative 
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evaluation  decisions  can  be  made  regarding  the  quality  of  the  lessons  and  the 
reliability  of  internal  quality  control  procedures.  Additional ly,  since  the 
actual  time  interval  for  learning  is  one  week,  threats  to  internal  validity  due 
to  maturation  and  history  are  minimal. 

Testing  effects  may  be  the  most  difficult  source  of  invalidity  to  overcome 
since,  ideally,  there  should  be  multiple  parallel  forms  of  lesson  pretests  and 
posttests.  These  effects  will  be  minimized  by  not  providing  feedback  to 
soldiers  on  how  they  performed  on  individual  items  of  the  respective  tests 
during  the  lessons. 


01  XI  02  03 

Where: 

01  *  Locator  Test,  XI  =  short  lessons  (12) 

Lesson  pretests  long  lessons  (4) 

02  =  Lesson  posttests 

03  *  Summative  posttest 
CLOZE  Test 
Attitude  survey 


Figure  2.  JSEP  preliminary  tryout  design. 


THE  JSEP  FULL  SCALE  TRYOUT 

The  requirements  for  Task  16,  Conduct  JSEP  Tryouts  at  Two  TRAD0C  and  Two 
F0RSC0M  Sites,  indicate  that  both  the  curriculum  and  management  plan  should  be 
ready  for  trials  at  this  stage  of  development.  Accordingly,  we  propose  to 
conduct  the  Task  16  tryouts  presently  scheduled  at  Fort  Leonard  Wood,  Fort 
Bliss,  Fort  Riley,  and  Fort  Lewis. 

Based  on  results  from  the  full  scale  tryout,  inferences  will  be  made 
regarding  potential  JSEP  effectiveness,  efficiency,  and  worth  to  the  Army 
mission.  At  the  time  of  the  Task  16  tryout,  all  JSEP  elements  will  be  ready 
for  operational  testing.  The  following  objectives  for  the  full  scale  tryout 
address  and  expand  the  evaluation  questions  posed  in  the  solicitation,  the 
proposal,  and  in  subsequent  communications  with  ARI. 

Objectives 


The  following  are  the  objectives  of  the  full  scale  tryout: 
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1.  Examine  the  extent  to  which  JSEP  offers  a  prerequisite  skills- 
oriented,  job-related,  computer-based  curriculum  in  accordance  with  the 
specif ications  outlined  in  Task  5.  Specific  items  include: 

o  diagnosis  of  job-related  skill  deficiencies  through 
administration  of  the  RCA-developed  tests, 

o  an  appropriate  instructional  prescription  for  each  soldier, 

o  a  self-paced  computer  based  instructional  delivery  system  with 
appropriate  support  materials  to  address  these  deficiencies,  and 

o  criterion-referenced  appraisal  of  whether  the  deficiencies  have  been 
successfully  eliminated. 

2.  Identify  the  characteristics  of  individuals  for  whom  JSEP  is  most 
effective.  Here  effectiveness  of  JSEP  is  defined  as  the  successful  elimination 
of  measured  deficiencies  through  appropriate  instruction.  The  intent  under¬ 
lying  this  objective  is  to  identify  soldier  characteristics  that  are  related  to 
the  mastery  of  PCs  in  the  instructional  program.  Such  variables  might  include 
amount  of  instruction  prescribed,  educational  attainment,  rank,  ethnicity,  and 
verbal  and  quantitative  abilities. 

3.  Investigate  the  extent  to  which  JSEP  contributes  to  job  performance, 
operationally  defined  here  as  reading  comprehension  (measured  by  a  CLOZE  test) 
of  an  appropriate  field  manual,  application  of  quantitative  and  verbal  skills 
of  a  job  related  nature,  post-JSEP  ratings  of  soldiers  by  their  supervisors 
and  commanders  (to  the  extent  this  data  is  available),  and  self-ratings  by  JSEP 
participants  after  they  have  returned  to  the  job. 

Measures  of  true  impact  on  job  performance  should  be  collected  on  a  much 
broader  basis  than  will  be  possible  before  this  contract  expires.  Even  during 
the  full  scale  tryout,  we  will  not  be  able  to  serve  enough  soldiers  to  make 
statistically  reliable  estimates  of  the  impact  of  JSEP  that  could  be  projected 
Army-wide. 

We  believe  that  full  scale  implementation  of  JSEP  will  require 
continuing  contractor  support  until  the  system  has  become  fully  operational. 
This  period  of  support  should  be  long  enough  to  permit  the  collection  of 
professionally  acceptable  data  to  thoroughly  assess  the  impact  of  JSEP  on  the 
Army  mission. 

4.  Explore  the  effects  that  learning  strategies  have  on  learning  in  JSEP. 
As  a  result  of  mastering  and  applying  certain  learning  strategies,  it  is 
hypothesized  that  soldiers  should  be  able  to  achieve  the  objectives  of  their 
lessons  more  efficiently  and  retain  the  concepts  and  intellectual  skills  they 
have  mastered  throughout  the  planned  period  of  measurement. 

5.  Investigate  the  effectiveness  and  efficiency  of  JSEP  system 
components.  The  principal  components  of  the  JSEP  instructional  system  include: 

o  the  computer  hardware  (PLATO,  T ICC  IT,  and  the  Hand-held 
Tutor), 


o  the  Locator  tests  that  identify  PC  deficiencies, 

o  the  JSEP  courseware  that  provides  prescribed 

instruction  directed  to  the  achievement  of  diagnosed  PC 
deficiencies, 

o  the  criterion  tests  that  indicate  mastery, 

o  the  management  information  system  to  store  and  retrieve 
relevant  information, 

o  the  JSEP  coordinators  who  provide  instructional 
support  when  requested, 

o  the  physical  environment  of  the  learning  centers,  and 

o  policies  and  procedures  that  provide  for  the  orderly 
conduct  of  JSEP. 

The  evaluation  procedures  for  each  of  these  will  be  outlined  in  the 
Methods  section  of  the  evaluation  plan  for  the  full  scale  tryout. 

6.  Infer  the  attitudes  of  soldiers  and  other  Army  personnel  toward  JSEP 
based  on  survey  instrument  responses  and  interviews.  Often  in  new  applications 
of  technology,  the  attitudes  toward  an  event  are  as  important  as  documented 
results. 

The  assessment  of  attitudes  toward  JSEP  by  its  participants  and  its 
"consumers"  will  be  a  vital  part  of  the  evaluation  plan.  The  primary  groups  of 
individuals  will  be  the  soldiers,  JSEP  coordinators,  ESOs,  and  available 
immediate  supervisors  of  the  soldiers  who  participate  in  the  tryout. 

7.  Investigate  the  potential  relationship  between  JSEP  participation  and 
scores  earned  on  tests  that  influence  career  goals.  Even  though  JSEP  is  not 
designed  primarily  to  develop  general  intellectual  abilities,  it  is  important 
to  investigate  whether  the  learning  of  job-related  PCs  could  have  some  degree 
of  transfer  value  to  performance  on  more  general  ability  measures.  In  this 
regard,  performance  measures  that  directly  affect  a  soldier's  career 
aspirations,  both  within  the  Army  and  in  the  civilian  world,  will  be  used  as 
summative  criterion  measures.  If  available,  these  will  include  the  GT 
(General  Technical  Area  Aptitude  subtest  of  the  ASVA6),  the  SQT  (Army  MOS 
Skills  Qualification  Test),  and  the  GED  (the  high  school  equivalency 
examination). 

An  acceptable  score  on  the  GT  qualifies  a  soldier  for  reenlistment  and 
promotion  in  the  Army.  While  we  will  be  able  to  accrue  data  regarding  GT  and 
GEO  performance,  subsequent  investigators  must  conduct  the  investigation  of  the 
relationship  between  JSEP  participation  and  SQT  performance  since  intensive 
follow-up  procedures  will  be  required. 

8.  Examine  the  validity  (concurrent  and  predictive)  of  the  Locator  Tests 
for  identifying  PC  deficiencies  and  for  prescribing  appropriate  instruction. 
Correlational  and  classification  studies  will  be  performed  on  the  Locator  Test 
and  the  lesson  pretests. 
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Properly  developed,  these  tests  should  measure  the  same  skills  (but 
with  different  reliabilities).  There  should  be  high  attenuated  correlations 
among  the  two  measures  if  they  are  to  be  used  for  the  purpose  of  diagnosing  PC 
deficiencies  and  for  routing  learners  into  appropriate  lessons. 

The  outcome  of  this  two  stage  test  sequence  should  predict,  within  certain 
confidence  intervals,  the  likelihood  of  passing  the  lesson  posttest. 

Therefore,  the  accuracy  will  be  investigated  with  which  the  Locator  Tests  are 
able  to  determine  needs  for  instruction  as  indicated  by  performance  on  the 
short  lesson  posttests  and  the  long  lesson  posttests. 

9.  Examine  the  potential  flexibility  of  JSEP  as  an  open-entry,  open- 
access  instructional  system.  A  self-instructional  system  can  offer  the 
opportunity  for  soldiers  to  enter  and  exit  the  JSEP  system  as  their  job  demands 
dictate.  Under  the  JSEP  management  plan,  the  education  center  could 
theoretically  accommodate  soldiers  any  time  of  day  or  night  and  provide  quality 
instruction.  The  feasibility  of  moving  toward  an  open-entry,  open-access 
instructional  system  will  be  explored  as  part  of  the  evaluation. 

10.  Assess  the  pre-service  and  in-service  training  needs  of  JSEP 
coordinators  related  to  the  curriculum  and  standard  operating  procedures.  A 
quasi  "task  analysis"  of  activities  will  be  performed  to  identify  the  tasks 
that  JSEP  coordinators  perform.  Each  of  the  tasks  will  be  rated  in  terms  of 
difficulty  and  need  for  pre-service  or  in-service  training.  At  least  one 
coordinator  at  each  site  should  be  from  the  education  centers  or  the 
prospective  coordinator  population  and  have  been  trained  by  FSU-Hazeltine. 
Interviews  with  JSEP  coordinators  and  ESOs  will  also  be  conducted  by  the  FSU- 
Hazeltine  team.  With  the  advice  of  the  coordinators  and  ESOs,  a  document  on 
Standard  Operating  Procedures  will  be  developed. 


JSEP  System  Components 


At  the  time  of  the  Task  16  tryout,  all  JSEP  System  components  will  have 
been  developed  and  will  have  undergone  at  least  an  internal  review  and  revision 
sequence.  The  following  elements  of  the  system  will  be  installed  at  the  four 
designated  tryout  sites: 


1.  Terminals: 


2.  Coordinators: 


3.  Soldiers: 


It  is  now  planned  that  TICCIT  systems  will  be 
installed  for  the  tryouts  at  Fort  Riley  and  Fort 
Lewis,  and  PLATO  systems  will  be  available  at  Fort 
Bliss  and  Fort  Leonard  Wood.  Each  installation  is  to 
have  20  terminals. 

There  will  be  at  least  one  trained  JSEP  coordinator 
on  duty  at  each  delivery  site  during  the  hours  the 
education  centers  are  customarily  open. 

Participants  will  be  drawn  from  a  population  of 
soldiers  newly  arrived  at  each  site  whose  GT  or 
Locator  scores  suggest  they  may  have 
prerequisite  skill  deficiencies  that  would  limit 
their  ability  to  master  certain  job  tasks. 


4.  Management  Information  System:  A  management  information  system  (MIS) 

will  have  been  developed  that  stores  student  back¬ 
ground  data,  student  test  and  lesson  performance 
data,  and  evaluation  information.  The  MIS  will 
also  provide  information  about  Locator  Test 
scores  and  job  task  statements  on  which  a 
soldier  is  predicted  to  have  difficulty.  Rules 
for  accessing  such  information  will  have  been 
formulated  prior  to  the  tryout. 

5.  Policies  and  Procedures:  A  code  of  Standard  Operating  Procedures  will 

be  developed  and  implemented  to  govern  the  flow  of 
soldiers  and  support  personnel  through  JSEP  during 
the  full  scale  tryout. 

6.  The  JSEP  Curriculum:  The  curriculum  will  consist  of  the  following 

components: 

(a)  Orientation  to  JSEP  which  will  include  an 
explanation  on  the  purpose  of  JSEP,  its 
structure,  operating  procedures,  and  instructions 
in  how  to  operate  the  computer, 

(b)  Locator  Test, 

(c)  Learning  strategies  orientation  with  prompts 
in  the  long  lessons, 

(d)  Short  lessons  for  all  PCs.  Each  lesson  will 
include  two  elements:  (1)  instruction 
pertaining  to  essential  rules,  operations,  and 
concepts  along  with  practice  exercises  to 
"revive"  prior  learning  of  the  PCs,  and  (2)  the 
lesson  posttest  (10-15  item  multiple  choice 
test). 

(e)  Long  lessons  on  selected  high  priority  PCs  with  a 
posttest  for  those  soldiers  who  do  not  pass  the 
posttest  of  the  short  lessons. 

(f)  Summative  posttest. 

(g)  Performance  test  consisting  of  a  CLOZE  reading 
test  on  an  appropriate  Soldier's  or  Technical 
manual.  Other  performance-like  measures  will  be 
explored  prior  to  the  tryout. 


JSEP  Process 

The  soldier  flow  through  the  JSEP  management  system  can  be  understood  by 
following  the  flowchart  presented  in  Figure  3.  Soldiers  will  take  the  Locator 
Test.  If  the  scores  on  the  Locator  predict  potential  deficiencies  in  PCs 


JSCP  Process 


required  for  learning  the  MOS  job  tasks,  soldiers  are  routed  to  the  prescribed 
short  lessons. 

They  go  first  to  the  designated  JSEP  short  lesson  and  receive  a 
"refresher"  review  of  the  material.  They  then  take  the  lesson  posttest. 

If  they  fail  the  posttest  after  the  short  lesson,  they  are  then  routed 
immediately  to  a  long  lesson  which  consists  of  in-depth  instruction  of  the  PC. 
The  long  lesson  also  contains  a  posttest.  If  soldiers  repeatedly  fail  the 
posttest  after  the  long  lesson,  they  will  then  be  instructed  to  go  to  the  JSEP 
coordinator  for  special  assistance. 

When  soldiers  have  completed  all  of  their  assigned  lessons  by  passing  all 
of  the  lesson  posttests,  they  will  take  a  summative  posttest  consisting  of 
approximately  3  items  randomly  drawn  from  each  of  the  posttests  of  the  lessons 
to  which  they  were  assigned.  If  they  pass  the  summative  posttest,  they  will  be 
declared  to  have  passed  JSEP.  Then,  they  will  take  a  performance  test 
that  is  intended  to  be  related  to  job  performance. 


Physical  Facilities 

Tryout  sites  will  be  configured,  to  the  fullest  extent  possible,  to 
resemble  an  implementation  site.  Since  there  are  many  variations  in  the  Army, 
it  will  be  difficult  to  predict  installation  or  operations  problems  for  all 
sites.  It  is  planned  that  a  special  room  will  be  designated  for  "noisy"  or 
disturbing  equipment  such  as  a  printer  or  mainframe.  We  hope  to  find  suitable 
large  areas  for  the  20  terminals  with  a  JSEP  coordinator's  desk  near  the 
entrance.  The  coordinator's  space  may  include  a  table,  chair,  a  work  table  for 
several  soldiers,  and  ancillary  print  materials  and  hand-held  computers  for 
specific  drill  purposes. 


METHOD:  THE  FULL  SCALE  TRYOUT 


The  Soldiers 


We  propose  to  identify  tryout  soldiers  from  the  pool  of  those  who  have 
taken  the  Locator  Test  prior  to  the  tryout  start-up  date  or  from  soldiers 
either  for  whom  GT  scores  are  available.  A  roster  of  "JSEP"  eligible"  soldiers 
will  be  generated  at  each  base  who  fall  within  Locator  or  GT  score  parameters. 

Those  soldiers  who  have  been  in  their  duty  station  for  more  than  three 
months  will  be  assigned  to  a  second  priority  status,  to  be  used  only  if  we  are 
unable  to  secure  a  sufficient  number  of  soldiers  for  the  tryout  from  the  first 
priority  group. 

From  the  first  priority  roster  described  above,  a  group  of  120  will  be 
identified  and  60  randomly  assigned  to  the  treatment  group  while  60  others 
will  be  assigned  to  the  no-treatment  (control)  group.  Command  support  will  be 
required  to  permit  the  selection  of  the  appropriate  population  for  the  tryout. 
A  special  effort  will  be  made  at  each  post  to  obtain  the  support  of  the 
commandant  for  the  period  of  the  tryout.  Past  experience  indicates  that  Army 
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priorities  are  often  changed  and  neither  the  potential  difficulty  in  obtaining 
true  J SEP  eligible  soldiers  nor  the  criticality  of  doing  so  can  be 
overestimated. 

Additional  soldiers  may  be  drawn  from  the  pool  or  from  the  second  priority 
pool  depending  on  the  degree  of  command  support  for  the  tryouts.  Thus  by 
combining  all  four  sites,  there  will  be  240  experimental  OSEP  soldiers  and  240 
control  soldiers.  Population  samples  of  this  size  will  provide  adequate 
statistical  power  with  which  to  use  inferential  statistics  in  the  analysis  of 
data. 

The  COs  of  the  JSEP  controls  will  also  be  sent  a  memorandum  notifying 
them  of  their  soldiers  Locator  Test  scores,  their  predicted  PC  deficiencies 
and  the  job  tasks  the  soldiers  may  encounter  difficulty  in  mastering  because 
of  predicted  PC  deficiencies.  The  COs  will  be  asked  to  release  the  members  of 
this  group  from  duty  to  take  the  JSEP  Pretest  Battery  which  consists  of  the 
assigned  lesson  pretests,  the  summative  posttest  Form  I,  the  GT,  the  Test  of 
Adult  Basic  Education  (TABE),  the  self-concept  as  a  learner  measure  (described 
below),  the  Learning  Strategies  Mastery  test,  and  the  performance  pretest.  We 
estimate  it  will  take  15  to  20  hours  to  complete  the  test  battery.  Five  weeks 
later,  we  will  also  request  that  control  group  soldiers  be  released  from  duty 
for  an  equal  period  of  time  to  complete  the  posttest  battery  which  is  an 
alternate  form  of  the  JSEP  Pretest  Battery.  The  use  of  a  control  group  will 
help  to  control  for  maturation  effects  due  to  time  on  the  job  and  informal 
learning. 

An  additional  comparison  group  will  consist  of  soldiers  currently 
enrolled  in  BSEP  who  will  take  the  same  pretest  and  posttest  as  the  JSEP 
controls  and  at  the  same  time  as  the  controls.  The  FSU-Hazeltine  team  will 
request  that  60  BSEP  II  soldiers  at  each  site  be  assigned  to  take  the  JSEP 
pretest  and  posttest  batteries.  The  principal  reason  for  including  the  BSEP 
II  comparison  group  is  to  address  the  issue  of  whether  JSEP  can  be  effective 
in  helping  individuals  qualify  for  promotion  or  reenlistment  through  the 
raising  of  GT  scores.  Likewise,  an  alternative  issue  of  whether  BSEP  II  is 
effective  in  helping  soldiers  master  PCs  is  also  addressed  through  such  a 
design. 

Procedures 

After  the  60  JSEP  experimental  soldiers  have  been  assigned  to  participate 
in  JSEP  at  each  site,  20  will  be  randomly  assigned  to  the  morning  session,  20 
to  the  afternoon,  and  20  to  the  evening  session.  When  the  soldiers  report  to 
the  education  center  at  their  assigned  times,  a  JSEP  coordinator  will  check 
them  in  and  assign  each  a  terminal  for  use  during  the  tryout. 

The  coordinator  will  conduct  an  orientation  session  which  will  include  a 
statement  of  the  purpose  and  structure  of  JSEP  and  a  demonstration  on  how  to 
log-on  and  use  the  computer.  Following  the  orientation,  the  soldiers  will  take 
the  following  instruments:  JSEP  Background  Survey,  prescribed  lesson  pretests, 
TA8E  (if  they  have  not  already  taken  it),  the  GT  (also  if  they  have  not  already 
taken  it),  the  Learning  Strategies  Mastery  Test,  the  Learning  Self-Concept 
Inventory,  and  the  CLOZE  pretest.  The  soldiers  will  then  engage  in  the 
Learning  Strategies  lesson  which,  according  to  present  estimates,  will 
encompass  about  20  hours  of  instruction. 


Following  the  learning  strategies  lesson,  the  soldiers  will  engage  in  the 
short  lessons  for  those  PCs  that  have  been  prescribed.  Should  soldiers  fail 
the  posttest  after  a  short  lesson,  they  will  be  assigned  to  a  corresponding 
long  lesson  which  treats  the  PC  in  greater  depth.  Here  the  JSEP  coordinator 
may  assist  the  individuals  in  reviewing  the  results  of  the  pretests  and 
posttest  so  that  subjects  gain  some  awareness  of  the  nature  of  their 
deficiencies.  Following  the  long  lesson,  soldiers  will  take  a  posttest 
followed  by  the  Long  Lesson  Questionnaire.  If  soldiers  fail  the  posttest  of  a 
long  lesson,  they  will  again  seek  the  assistance  of  the  coordinator  for  review 
of  the  posttest  and  to  recycle  back  through  the  lesson,  or  parts  of  it. 

When  soldiers  complete  their  assigned  lessons  by  passing  the  posttests, 
they  will  then  take  a  summative  posttest  (final  exam).  If  the  subjects  pass 
their  summative  posttest,  they  will  take  JSEP  Posttest  Battery  consisting  of 
the  CLOZE  Test,  Learning  Strategies  Mastery  Test,  the  Self-Concept  as  a  Learner 
Test,  TABE,  6T,  and  the  JSEP  General  Questionnaire.  Should  soldiers  fail  their 
final  exam,  they  will  again  seek  the  assistance  of  the  JSEP  coordinator  to 
review  the  results  of  the  test  and  to  plan  further  instruction.  This  may 
consist  of  one-on-one  instruction,  a  second  pass  through  short  lessons  or  long 
lessons,  or  a  combination  of  these  followed  by  a  second  attempt  at  the 
summative  posttest.  All  soldiers  before  they  return  to  duty  will  take  the  JSEP 
Posttest  Battery. 

Should  soldiers  be  unable  to  complete  their  assigned  lessons  by  the  end  of 
the  tryout  period,  they  will  be  required  to  take  a  tailored  summative  post¬ 
test  over  their  assigned  lessons  in  the  last  week.  Pass  and  fail  rates  will 
be  carefully  noted  in  these  circumstances. 


The  JSEP  Instruments 


Some  of  the  instruments  described  in  Part  I  on  the  preliminary  tryout  will 
be  refined  and  used  in  the  full  scale  tryout.  Elaboration  of  the  instruments 
described  herein  will  chiefly  concern  the  instruments  that  have-not  been 
detailed  earl ier. 

1.  JSEP  Soldier  Background  Information  Survey.  This  will  be  the  same 
instrument  as  used  in  the  prel iminary  tryout. 

2.  Resource  Utilization  Survey:  JSEP  Learning  Center  Operation.  This 
instrument  will  be  similar  to  the  instrument  used  in  the  preliminary  tryout  in 
terms  of  the  kinds  of  information  sought,  but  it  will  extend  over  the  entire  5 
week  JSEP  cycle.  There  may  also  be  some  elaboration  to  record  the  use  of 
support  media  and  services.  The  respective  ESOs  will  be  consulted  in 
developing  the  final  version  of  the  instrument.  This  instrument  will  be 
completed  each  day  by  the  JSEP  coordinator. 

3.  JSEP  Learning  Outcome  Form.  The  full  scale  tryout  will  gather  the 
same  information  as  the  preliminary  tryout  and  will  be  administered  on-line. 

4.  Long  Lesson  Questionnaire.  This  will  be  the  same  instrument  as  was 
used  in  the  preliminary  tryout  and  will  be  administered  as  soldiers  complete 
the  lesson. 
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5.  JSEP  General  Questionnaire.  This  will  be  the  same  instrument  as  is 
used  in  the  prel iminary  tryout,  and  will  be  administered  as  soldiers  complete 
their  set  of  assigned  lessons,  or  in  the  fifth  week  of  the  JSEP  tryout  by  those 
soldiers  who  have  not  completed  their  lessons. 

6.  JSEP  Job  Performance  Test.  One  of  the  skills  soldiers  should  possess 
is  the  ability  to  comprehend  their  manuals.  Since  JSEP  offers  a  curriculum 
that  seeks  to  develop  cognitive  capabilities  (PCs)  as  opposed  to  psychomotor 
abilities,  a  proximal  job  performance  test  should  be  one  that  is  particularly 
sensitive  to  cognitive  development.  We  believe  there  is  a  significant  amount 
of  validity  in  the  proposition  that  gaining  intellectual  skills  and  mastering 
concepts  and  rules  in  JSEP  should  transfer  to  the  capability  for  comprehending 
manuals  with  greater  accuracy. 

The  characteristics  of  a  job  performance  test  should  include  the  following 
considerations: 

1.  It  should  be  job-related, 

2.  it  should  be  amenable  to  the  development  of  multiple 
parallel  forms, 

3.  it  should  be  easy  to  administer  and  score, 

4.  it  should  possess  adequate  reliability, 

5.  it  should  require  not  more  than  one  hour  to 
complete,  and, 

6.  it  should  yield  results  that  are  quantifiable  to 
allow  comparisons  between  groups  of  soldiers. 

The  CLOZE  procedure  (Harris  &  Sipy,  1980;  Klare,  Sinaiko,  &  Stolurow, 

1972)  meets  the  above  conditions.  The  CLOZE  procedure  requires  readers  to 
supply  missing  words  omitted  from  a  passage  (such  as  every  fifth  word).  It 
is  based  on  the  assumption  that  a  reader  who  is  able  to  understand  the 
structure  and  content  of  a  written  message  should  be  able  to  fill  in  words 
removed  from  it. 

A  prototype  CLOZE  procedure  was  used  to  evaluate  the  outcomes  of  the  BSEP 
II  program  at  Fort  Lewis  with  promising  results  (Banner,  Ridge,  Hewitt,  & 
Christ,  1982).  Here,  one  passage  was  randomly  selected  from  each  of  two  field 
manuals  Soldiers  Manual,  MOS  llBlO-Infantryman-llBlO,  FM  7-11B1/2,  and 
Soldier'sTianua),  MOS  13B/Cannon  Crewman,  Skill  Level  l~~and  2. 

The  FSU-Hazeltine  team  will  follow  upon  the  work  initiated  by  Banner  et 
al.  (1982)  and  produce  a  CLOZE  Test  that  will  be  applicable  across  a  wider 
array  of  MOS.  An  attempt  will  be  made  to  include  graphic  and  quantitative 
components  as  well.  The  CLOZE  Test  will  be  pilot  tested  during  the  preliminary 
tryout.  Two  forms  of  the  test  will  be  constructed  for  use  as  pretest  and 
posttest  measures. 


8.  JSEP  Job  Performance  Measure  (JPM).  In  addition  to  the  possible  use 
of  Enlisted  Efficiency  Reports  (EERs)  to  assess  the  influence  of  JSEP  on  job 
performance,  the  FSU-Hazeltine  team  will  develop  a  job  performance  measure 
based  on  task  statements.  Since  the  JSEP  curriculum  evolved  from  the  synthesis 
of  job  task  indicator  statements,  a  valid  0PM  to  test  the  transfer  of  knowledge 
and  intellectual  skills  to  job  task  performance  should  encompass  the  use  of 
appropriate  task  statements.  As  envisioned  at  the  present  time,  an  NCO  will 
complete  a  checklist  consisting  of  task  statements  that  soldiers  might  have 
difficulty  performing  given  their  Locator  Test  scores. 

At  the  present  time  we  plan  to  develop  a  JPM  with  25-30  MOS  related 
critical  task  statements  derived  from  the  RCA  task  analysis.  The  NCO  will  rate 
each  task  statement  in  terms  of  whether  or  not  the  soldier  performs  the  task  at 
an  acceptable  proficiency  level.  Here,  acceptable  proficiency  will  be  defined 
in  terms  of  the  way  the  NCO  defines  "acceptable". 

It  will  be  highly  desirable  to  collect  these  data  from  the  units  for  JSEP 
experimentals,  JSEP  controls,  and  for  BSEP  II  populations  three  months  after 
the  JSEP  Tryout  has  been  completed.  Whether  the  data  can  be  collected  depends 
on  cooperation  from  each  of  the  tryout  sites. 

9.  Learning  Strategies  Mastery  Test.  A  test  will  be  designed  and 
developed  to  assess  the  degree  to  which  soldiers  possess  certain  learning 
strategies.  The  instrument  may  consist  of  a  study  skills  inventory,  selected 
memory  performance  tasks,  test  taking  skills  measures,  positive  self  image 
measures,  and  selected  problem  solving  tasks. 

10.  Self  Concept  as  a  Learner  Inventory.  Prior  to  the  full  scale  tr'»ut, 
the  FSU-Hazeltine  team  will  identify  instruments  that  assess  self  concept  as  a 
learner.  Such  instruments  assess  dimensions  such  as  confidence  in  learning  new 
material,  and  the  belief  in  the  ability  to  attain  educational  goals.  If  it  is 
shown  that  JSEP  has  an  impact  on  self  concept  as  a  learner,  there  would  be  a 
compelling  justification  for  institutionalizing  JSEP  since  this  would  provide 
evidence  that  JSEP  can  help  soldiers  "be  all  they  can  be." 

Evaluation  Design 

The  evaluation  design  for  the  full  scale  tryout  can  be  described  as  a 
randomized  pretest-posttest  experimental  and  comparison  group  design  where  JSEP 
samples  drawn  at  random  from  Locator  Test  rosters.  BSEP  II  subjects  are  drawn 
from  classes  beginning  approximately  six  months  after  the  start  of  Phase  III. 
JSEP,  BSEP  II,  and  control  group  soldiers  will  take  the  same  pretest,  post¬ 
tests,  and  undergo  the  same  follow-up  observations.  Such  a  design  is  diagram¬ 
med  in  Figure  4. 

Learning  strategies  evaluation.  The  evaluation  of  the  contribution  of 
learning  strategies  to  JSEP  is  approached  from  two  perspectives: 

1.  proximal  effect,  which  is  the  growth  in  certain  capabilities,  and, 

2.  the  impact  of  learning  strategies  on  external  criteria. 

Both  of  these  perspectives  are  required  to  determine  the  cost-effective¬ 
ness  of  the  learning  strategies  component. 
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Figure  4.  JSEP  full  scale  tryout  evaluation  design. 


Proximal  effects.  The  Learning  Strategies  Mastery  Test  Forms  I  and  II 
will  be  administered  in  a  pre-posttest  paradigm  to  derive  gain  scores  for  four 
evaluation  groups  (JSEP  experimentals  with  learning  strategies  (Randomized 
group  1),  JSEP  experimentals  without  learning  strategies  (Randomized  group  2), 
JSEP  controls  (Randomized  group  3),  and  BSEP  II  (Comparison  group  1)).  The 
gain  scores  of  respective  groups  will  be  compared  through  analysis  of 
covariance  with  the  pretest  scores  used  as  the  covariates.  All  individuals  in 
the  JSEP  experimental  group  will  have  had  some  minimal  exposure  to  the 
treatment. 

Impact  of  mastery  of  learning  strategies.  Learning  strategies  theories 
hold,  according  to  Oansereau  (1983)  and  McCombs  (1983),  that  learning 
strategies  capabilities  will  influence  the  rate  of  learning,  the  ability  to 
recall  concepts  and  rules,  and  the  ability  to  solve  problems.  The  learning 
strategies  curriculum  in  JSEP  consists  of  first  being  exposed  to  a  learning 
strategies  orientation  in  which  soldiers  learn  "learning"  concepts  and  skills. 
These  are  applied  and  reinforced  in  the  ensuing  long  lessons  to  which  the 
soldiers  are  assigned.  Gains  in  learning  strategy  capability  should  manifest 
themselves  in  performance  on  the  summative  posttest  (a  recall  and  recognition 
test)  and  in  lesson  completion  time. 

To  isolate  the  effects  of  learning  strategies  mastery,  two  subgroups  of 
the  experimental  JSEP  group  will  be  formed: 

1.  those  who  have  the  learning  strategies  orientation  experience  plus 
four  long  lessons  with  intensive  learning  strategy  reinforcers,  and 

2.  those  who  do  not  have  the  learn-ng  strategies  orientation  but  are 
assigned  the  same  four  long  lessons  in  addition  to  their  prescribed 
lessons  based  on  the  Locator  Test  and  lesson  pretests. 

Even  though  the  latter  group  may  have  learning  strategy  reinforcers 
embedded  within  existing  lessons,  without  the  initial  learning  strategy  mastery 
and  skill  development  acquired  during  orientation,  the  effect  of  the  prompts 
within  JSEP  lessons  may  be  negligible. 

Soldiers  will  be  randomly  assigned  to  one  of  these  two  groups;  120  will 
have  the  JSEP  learning  strategies  orientation  plus  reinforcement,  and  120  will 
not  have  the  JSEP  learning  strategies  orientation.  Both  groups  will  be 
compared  in  terms  of  performance  on  the  summative  posttest,  CLOZE  Test,  lesson 
completion  rate,  and  job  task  performance.  Analysis  of  covariance  will  be  used 
to  partial  out  the  effects  of  Learning  Strategies  Mastery  pretest  scores. 
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APPENDIX  A 

Long  Lesson  Questionnaire 


Long  Lesson  Questionnaire 


Lesson  Title: 

I.  Read  each  question  below.  Circle  the  number  that  comes  closest  to 
your  opinion. 


1.  Were  the  words  in  the  lesson  easy  or  hard  to  read? 


1 

2 

3 

easy 

neither 

hard 

2. 

Were  the  pictures 

in  the  lesson 

easy  or  hard  to  figure  out? 

1 

2 

3 

easy 

neither 

hard 

3. 

Were  the  pictures 

in  the  lesson 

helpful  or  not  helpful? 

1 

2 

3 

helpful 

neither 

not  helpful 

4. 

Were  the  practice 
helpful . 

exercises  in 

the  lesson  helpful  or  not 

1 

2 

3 

helpful 

neither 

not  helpful 

5. 

Were  the  messages 

for  wrong  answers  helpful  or  not  helpful? 

1 

2 

3 

helpful 

neither 

not  helpful 

6.  Were  the  test  questions  at  the  end  of  the  lesson  easy  or 
hard? 


7.  Was  the  arrangement  of  the  material  in  the  lesson  easy  or 


hard  to  follow? 

1  2  3 

easy  neither  hard 

8.  Overall,  was  the  lesson  interesting  or  boring? 

1  2  3 

interesting  neither  boring 

9.  Overall,  was  the  lesson  easy  or  hard? 

1  2  3 

easy  neither  hard 

10.  Would  you  have  preferred  to  use  a  printed  workbook  rather 

than  a  computer  for  this  lesson? 

1  2  3 

yes  doesn't  matter  no 

11.  Would  you  have  preferred  to  learn  this  lesson  from  an 

instructor  rather  than  a  computer? 

1  2  3 

yes  doesn't  matter  no 


II.  Read  each  question.  Write  your  answers  in  the  space  after  each 
question. 

1.  What  do  you  think  was  the  best  part  of  the  lesson? 


2.  What  do  you  think  was  the  worst  part  of  the  lesson? 
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(a)  What  do  you  think  was  the  hardest  part  of  the  lesson? 


(b)  How  do  you  think  it  could  have  been  made  more  effective? 


(a)  Do  you  think  the  short  review  lesson  that  you  took  before 
this  lesson  should  be  changed? 

_ YES  _ NO 

(b)  If  yes,  how  should  it  be  changed? 


JSEP  General  Questionnaire 


Read  each  statement  below.  Circle  the  number  that  comes  closest 
to  your  opinion. 


1.  It  was  easy  for  me  to  learn  to  use  the  computer. 


1  2  3  4  5 

strongly  agree  uncertain  disagree  strongly 

agree  disagree 

2.  Using  the  computer  for  four  hours  at  a  time  was  too  long. 

1  2  3  4  5 

strongly  agree  uncertain  disagree  strongly 

agree  disagree 


3.  I  think  the  instructor  in  the  educational  center  should  teach 
the  lessons  instead  of  the  computer. 


1 

strongly 

agree 


2  3 

agree  uncertain 


4  5 

disagree  strongly 
disagree 


4.  The  Hand-held  Tutor  helped  me  to  learn. 


1  2  3 

strongly  agree  uncertain 

agree 


4  5 

disagree  strongly 
disagree 


5.  I  think  computer  lessons  should  be  used  only  to  practice  what 
we  learn  from  an  instructor. 


1 

2 

3 

4 

5 

strongly 

aqree 

agree 

uncertain 

disagree 

strongly 

disagree 

6.  I  think  written  assignments  should  be  used  along  with  computer 
lessons. 


1  2  3  4  5 

strongly  agree  uncertain  disagree  strongly 
agree  disagree 


7.  The  lesson  about  mood  management  helped  me  control  my 
mood  during  the  other  OSEP  lessons. 


1  2  3  4  5 

strongly  agree  uncertain  disagree  strongly 
agree  disagree 


8.  I  think  I  will  use  what  I  learned  about  mood  management  in 
other  situations. 


1 

Strongly 

agree 

2 

agree 

3 

uncertain 

4 

disagree 

5 

strongly 

disagree 

The  lessons  I  studied 

will  help  me  in  my  MOS. 

1 

strongly 

agree 

2 

agree 

3 

uncertai n 

4 

disagree 

5 

strongly 

disagree 

Overall, 

the  JSEP  program  was  well- 

managed. 

1 

strongly 

agree 

2 

agree 

3 

uncertain 

4 

disagree 

5 

strongly 

disagree 

I  would  be  willing  to  take  more  JSEP 
computer  if  they  were  offered  during 

lessons  offered  on  a 
on-duty  hours. 

1 

strongly 

agree 

2 

agree 

3 

uncertain 

4 

disagree 

5 

strongly 

disagree 

12.  I  would  be  willing  to  take  more  JSEP  lessons  offered  on  a 
computer  if  they  were  offered  during  off-duty  hours. 


1  2  3  4  5 

strongly  agree  uncertain  disagree  strongly 
agree  disagree 


13.  I  think  my  unit  commander  would  be  willing  to  release  me 
from  duty  to  take  JSEP  lessons. 


1  2  3  4  5 

strongly  agree  uncertain  disagree  strongly 
agree  disagree 


14.  I  think  JSEP  lessons  will  help  me  read  and  understand  the 
publications  I  use. 


1  2  3  4  5 

strongly  agree  uncertain  disagree  strongly 

agree  disagree 

15.  The  skills  I've  learned  in  JSEP  will  help  me  advance  to  a 
higher  grade/rank  in  the  Army. 

1  2  3  4  5 

strongly  agree  uncertain  disagree  strongly 

agree  disagree 


16.  I  think  JSEP  should  be  included  as  an  educational  program 
offered  by  the  Educational  Center. 


1  2  3  4  5 

strongly  agree  uncertain  disagree  strongly 
agree  disagree 


Read  the  following  list  of  reasons  for  taking  future  JSEP  lessons. 
Put  a  check  on  the  line  before  each  one  that  would  be  an  important 
reason  for  you  to  take  more  JSEP  lessons. 

_  for  self  improvement 

_  for  general  knowledge 

_  to  increase  job  proficiency 

_  to  pass  the  SQT 

_  to  raise  ASVAB  (GT)  score  for  reenlistment 

_  to  qualify  for  a  different  MOS 

to  obtain  a  GED  certificate 
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APPENDIX  C 

Soldier  Background  Information  Survey 


33 


KO. 


First  Week 


Second  Week 
Observer 


Resource  Utilization  Survey: 
Learning  Center  Operation 


The  following  data  will  be  recorded  by  observer  from  FSU-Hazeltine. 

1.  Number  of  troops  M  T  W  Th  F 
attending  (FTE) 

Group  I  PLATO 

(n  =  8)  _  _  _  _  _ 

Group  II  TICCIT 

(n  -  8)  _  _  _  _ 


2.  Computer  down-  M  T  W  Th  F 

times  (Please 

record  time  in  From  _  _  _  _  _ 

appropriate  To  _  _  _  _  _ 

places). 

From 

To 


From 

To 


3.  Environmental  M  T  W  Th  F 

distractions 
(Please  describe 
below) 

a)  From 

To 


b)  From 
To 

c)  From 
To 
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APPENDIX  E 

Learning  Outcome  Form 


JSEP  Learning  Outcome  Form 


Name _ _ 

Soldier  _ 

cohort  1  n  m  nr 

1.  Locator  Scores 

2.  Lesson  Performances 


V 

M 


3.  Final  exam  pretest  score 

4.  Final  exam  posttest  score 

5.  RCA  criterion  test 
pretest  score 

6.  RCA  criterion  test 
posttest  score 

7.  CLOZE  test 
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